{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set tf 1.x for colab\n",
    "%tensorflow_version 1.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# MNIST digits classification with Keras\n",
    "\n",
    "We don't expect you to code anything here because you've already solved it with TensorFlow.\n",
    "\n",
    "But you can appreciate how simpler it is with Keras.\n",
    "\n",
    "We'll be happy if you play around with the architecture though, there're some tips at the end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/mnist_sample.png\" style=\"width:30%\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import accuracy_score\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "print(\"We're using TF\", tf.__version__)\n",
    "import keras\n",
    "print(\"We are using Keras\", keras.__version__)\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"../..\")\n",
    "import keras_utils\n",
    "from keras_utils import reset_tf_session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Look at the data\n",
    "\n",
    "In this task we have 50000 28x28 images of digits from 0 to 9.\n",
    "We will train a classifier on this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import preprocessed_mnist\n",
    "X_train, y_train, X_val, y_val, X_test, y_test = preprocessed_mnist.load_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# X contains rgb values divided by 255\n",
    "print(\"X_train [shape %s] sample patch:\\n\" % (str(X_train.shape)), X_train[1, 15:20, 5:10])\n",
    "print(\"A closeup of a sample patch:\")\n",
    "plt.imshow(X_train[1, 15:20, 5:10], cmap=\"Greys\")\n",
    "plt.show()\n",
    "print(\"And the whole sample:\")\n",
    "plt.imshow(X_train[1], cmap=\"Greys\")\n",
    "plt.show()\n",
    "print(\"y_train [shape %s] 10 samples:\\n\" % (str(y_train.shape)), y_train[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# flatten images\n",
    "X_train_flat = X_train.reshape((X_train.shape[0], -1))\n",
    "print(X_train_flat.shape)\n",
    "\n",
    "X_val_flat = X_val.reshape((X_val.shape[0], -1))\n",
    "print(X_val_flat.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one-hot encode the target\n",
    "y_train_oh = keras.utils.to_categorical(y_train, 10)\n",
    "y_val_oh = keras.utils.to_categorical(y_val, 10)\n",
    "\n",
    "print(y_train_oh.shape)\n",
    "print(y_train_oh[:3], y_train[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# building a model with keras\n",
    "from keras.layers import Dense, Activation\n",
    "from keras.models import Sequential\n",
    "\n",
    "# we still need to clear a graph though\n",
    "s = reset_tf_session()\n",
    "\n",
    "model = Sequential()  # it is a feed-forward network without loops like in RNN\n",
    "model.add(Dense(256, input_shape=(784,)))  # the first layer must specify the input shape (replacing placeholders)\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(256))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can look at all layers and parameter count\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we \"compile\" the model specifying the loss and optimizer\n",
    "model.compile(\n",
    "    loss='categorical_crossentropy', # this is our cross-entropy\n",
    "    optimizer='adam',\n",
    "    metrics=['accuracy']  # report accuracy during training\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and now we can fit the model with model.fit()\n",
    "# and we don't have to write loops and batching manually as in TensorFlow\n",
    "model.fit(\n",
    "    X_train_flat, \n",
    "    y_train_oh,\n",
    "    batch_size=512, \n",
    "    epochs=40,\n",
    "    validation_data=(X_val_flat, y_val_oh),\n",
    "    callbacks=[keras_utils.TqdmProgressCallback()],\n",
    "    verbose=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Here're the notes for those who want to play around here\n",
    "\n",
    "Here are some tips on what you could do:\n",
    "\n",
    " * __Network size__\n",
    "   * More neurons, \n",
    "   * More layers, ([docs](https://keras.io/))\n",
    "\n",
    "   * Other nonlinearities in the hidden layers\n",
    "     * tanh, relu, leaky relu, etc\n",
    "   * Larger networks may take more epochs to train, so don't discard your net just because it could didn't beat the baseline in 5 epochs.\n",
    "\n",
    "\n",
    " * __Early Stopping__\n",
    "   * Training for 100 epochs regardless of anything is probably a bad idea.\n",
    "   * Some networks converge over 5 epochs, others - over 500.\n",
    "   * Way to go: stop when validation score is 10 iterations past maximum\n",
    "     \n",
    "\n",
    " * __Faster optimization__\n",
    "   * rmsprop, nesterov_momentum, adam, adagrad and so on.\n",
    "     * Converge faster and sometimes reach better optima\n",
    "     * It might make sense to tweak learning rate/momentum, other learning parameters, batch size and number of epochs\n",
    "\n",
    "\n",
    " * __Regularize__ to prevent overfitting\n",
    "   * Add some L2 weight norm to the loss function, theano will do the rest\n",
    "     * Can be done manually or via - https://keras.io/regularizers/\n",
    "   \n",
    "   \n",
    " * __Data augmemntation__ - getting 5x as large dataset for free is a great deal\n",
    "   * https://keras.io/preprocessing/image/\n",
    "   * Zoom-in+slice = move\n",
    "   * Rotate+zoom(to remove black stripes)\n",
    "   * any other perturbations\n",
    "   * Simple way to do that (if you have PIL/Image): \n",
    "     * ```from scipy.misc import imrotate,imresize```\n",
    "     * and a few slicing\n",
    "   * Stay realistic. There's usually no point in flipping dogs upside down as that is not the way you usually see them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
